Integration of Speech & Video: Applications for Lip Synch: Lip Movement Synthesis & Time Warping

Throughout the past several decades, much research has been done in the area of signal processing. Two of the most popular areas within this field have been applications for speech recognition and image processing. Due to these extended efforts, today there are systems that can accurately recognize and transcribe the daily television news programs that are broadcast to our homes. There are also systems that can accurately locate and track the faces within those same news programs. Recently, a new field has emerged which focuses on combining the disciplines of speech recognition and image processing in a practical way. This interest has sparked research in a broad range of new application areas, including: • Enhancement of speech recognition via visual cues • Animation of interactive talking agents to facilitate human-computer interaction • Synchronization of audio and video tracks in the film industry • Correction of lip movement for films dubbed into a foreign language and in video conferencing This paper will discuss some of the current efforts in integrating speech and video in a practical way. It will briefly discuss some image processing methods for extraction of lip coordinates and lip features from video. The focus of this investigation, however, is the use of speech recognition and other signal processing techniques in the development of two practical systems: one for lip movement synthesis and the other for lip synchronization. First will be the use of a speech recognition technique (Viterbi forced alignment) to segment the lip features extracted from video on a phoneme by phoneme basis. Once the video features are segmented, appropriate models can then be created to link the audio and video features together. Effective models can be built on a very limited amount of training data. Next will be the development and description of a system for the creation of synthetic lip features based on information contained in the speech recognizer output and the models discussed earlier. These features can then be used to automatically generate accurate and believable synthetic lip movements that correspond to any audio speech waveform. Also, a separate system to automatically synchronize lip motion and voice is under development. This system is based on dynamic time warping techniques on the output of the speech recognizer and the models that relate the audio and video features. Finally, there will be some discussion about the methods for performance evaluation of such systems, as well as some ideas for future research in this and other areas of multimodal signal processing.